Color Filter Array, Imaging Device, and Image Processing Unit

ABSTRACT

A color filter array includes a plurality of filters, each having one of a plurality of types of spectral sensitivity and being disposed at the location of a corresponding one of a plurality of pixels. The filters of a predetermined type selected from among the plurality of types are arranged at the locations of the pixels in a checkered pattern, and the filters of some or all of the other types are randomly arranged at the pixel locations at which the filters of the predetermined type are not present.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese PatentApplication JP 2006-069541 filed in the Japanese Patent Office on Mar.14, 2006, the entire contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a color filter array, an imagingdevice, and an image processing unit used for an image capturingapparatus, such as a digital camera using a solid-state imaging device.

2. Description of the Related Art

Single-plate color image capturing apparatuses that include an imageprocessing unit are known. In the image capturing apparatuses, a colorfilter is bonded to each of a plurality of pixels of a single-platesolid-state imaging device and the image processing unit allocates allcolors to the position of each pixel using a mosaic image of colorscaptured by the imaging device.

In such image capturing apparatuses using a single-plate solid-stateimaging device, only a single spectral sensitivity is obtained.Accordingly, in general, to obtain a color image, different colorfilters are bonded to a plurality of pixels so as to be arranged in aspecific pattern. In the captured image, each pixel provides only onecolor. Therefore, in terms of colors, a mosaic image is generated.However, by interpolating the color of a pixel using color informationthat can be obtained from the adjacent pixels, an image in which eachpixel has a full color can be generated. Such an interpolating processis referred to as a “color separation process” or a “demosaic process”.

For most of the single-plate color image capturing apparatuses, as acolor filter arrangement, a Bayer array format described in Bryce E.Bayer, U.S. Pat. No. 3,971,065 entitled “COLOR IMAGING ARRAY” is used.As shown in FIG. 1, in this arrangement, G color filters are arranged ina checkered pattern so that the density of the G color filters is twicethat of R color filters or B color filters. Accordingly, in most methodsof interpolating a color of a pixel, a G color having a large amount ofinformation and a strong relation to the brightness of light isinterpolated to all the pixels first, and subsequently, R and B colorsare interpolated using the G color as a reference.

For example, in the method described in Ozawa, Akiyama, Satoh, Nagaharaand Miura, U.S. Pat. No. 4,716,455 entitled “Chrominance SignalInterpolation Device for a Color Camera,” on ground that a ratio of alow-frequency component of each color in a local region is substantiallyconstant, G color is allocated to all of the pixels. Thereafter, anaverage of a ratio of R color to G color and an average of a ratio of Bcolor to G color of the adjacent pixels are multiplied by the G color ofa pixel of interest. Thus, unknown color components are estimated.

In addition, in this interpolation method, on ground that the G colorfilters are arranged in a checkered pattern, the resolutions in ahorizontal direction and a vertical direction can be increased.

For example, Japanese Patent No. 2931520 entitled “Color SeparationCircuit of Single-plate Color Video Camera” describes a technique inwhich a correlation value at the position of an interpolated pixel in ahorizontal direction or in a vertical direction is computed. Twointerpolation values obtained by processing means appropriate when thecorrelation in the horizontal direction is strong and processing meansappropriate when the correlation in the vertical direction is strong aremixed using the correlation value.

According to the above-described techniques, in a single-plate colorimage capturing apparatus, the R, G, and B colors can be allocated topositions of all the pixels for high-resolution display.

However, since each color is discretely sampled, aliasing (overlap of ahigh-frequency component with lower frequency components) occurs when acaptured image contains a high-frequency component having a frequencyhigher than the Nyquist frequency, and therefore, a color different froman original color is estimated.

This color is referred to as a “false color”. The false color isnoticeable when a color filter arrangement in which color filters areregularly arranged is used, since a significant overlap occurs in aspecific spatial frequency range. Once a false color is generated, itcannot be determined whether a color is an original color having anoriginal low frequency or the false color caused by the overlap of ahigh-frequency component. Therefore, the false color cannot be removedby using a frequency filter.

Accordingly, to reduce the occurrence of a false color, knownsingle-plate color image capturing apparatuses need to include anoptical low-pass filter disposed in front of the imaging device so as toremove a high-frequency component in advance. However, the opticallow-pass filter does not have a sharp cut-off capability for the Nyquistfrequency. Therefore, if the single-plate color image capturingapparatuses attempt to completely prevent the occurrence of the falsecolor, low-frequency components having a frequency lower than theNyquist frequency could also be cut off.

In addition, in the Bayer arrangement, the Nyquist frequency of the Rsignal or the B signal is lower than that of the G signal. Accordingly,an optical low-pass filter suitable for the Nyquist frequency of the Rchannel or the B channel decreases the resolution of the G channel.

In practical applications, since a decrease in resolution is notallowed, complete removal of the false color is difficult. Furthermore,the installation of an optical low-pass filter prevents miniaturizationand cost reduction of the image capturing apparatus.

Additionally, in contrast to the Bayer arrangement of three RGB colorfilters, the fidelity and the dynamic range of colors can be increasedby using a filter arrangement of four colors or more.

For accurate color reproduction, a method using a large number offilters each transmitting light only in a narrow wavelength range, asshown in FIG. 2A, is more suitable than a method using a small number offilters each transmitting light in a wide wavelength range, as shown inFIG. 2C.

For an increase in the dynamic range, a method using a plurality offilters that have different transmittances but transmit light in thesame wavelength range, as shown in FIG. 2B, is more suitable than amethod using the filters shown in FIG. 2C.

However, the Bayer arrangement is still widely used. This is because asthe number of pixels for one color is decreased, the resolution of thatcolor channel deteriorates, and therefore, a false color easily occurs.

To solve this problem, technology has been, invented in which theregularity of the color filter arrangement is reduced in order to reducethe occurrence of a false color.

More precisely, this technology solves the following problem. That is, afalse color is visually noticeable and removal of the false color isdifficult if most of the false colors occur in a specific spatialfrequency range.

Similarly, as used herein, the reduction in the occurrence of a falsecolor refers to the reduction in the occurrence of a false colorconcentrated in a specific spatial frequency range.

In a pseudo-random Bayer arrangement introduced by FillFactory, Belgium,(this document is available athttp://www.fillfactory.com/htm/technology/htm/rgbfaq.htm), G colorfilters are arranged in a checkered pattern. In addition, at positionsother than those of the G color filters, R and B color filters arepseudo-randomly arranged. This arrangement is referred to as a“three-color G-checkered pseudo-random arrangement”.

Additionally, Japanese Unexamined Patent Application Publication No.2000-316169 describes a six-color random arrangement in which four sidesor four corners of a pixel of interest are adjacent to filters of sixcolors.

Furthermore, Mutze, Ulrich, Dr., EP Patent Publication No. 0,804,037entitled “Process and system for generating a full color image ormultispectral image from the image data of a CCD image sensor with amosaic color filter” describes an arrangement including a five-color3-by-3 repetition pattern and a pseudo-random pattern.

All of the above-described arrangements include a random pattern. Inaddition, the two arrangements described in Japanese Unexamined PatentApplication Publication No. 2000-316169 and EP Patent Publication No.0,804,037, (A2) employ filters of more than three colors.

Because of the random pattern in the arrangements, the false color isdispersed in a variety of spatial frequency ranges, and therefore, thefalse color is not noticeable. In addition, the increase in the numberof filters improves the dynamic range and the performance of the colorreproduction.

However, although the pseudo-random Bayer arrangement introduced byFillFactory has a pseudo-random pattern, only the positions at which afalse color occurs in the spatial frequency range are slightly differentfrom those in the Bayer arrangement. This is because the frequency ofthe repetition is low. Therefore, in practice, the pseudo-random Bayerarrangement reduces the occurrence of a false color little. In addition,since the pseudo-random Bayer arrangement is a three-color filterarrangement, the performance of color reproduction and the dynamic rangeare substantially the same as those of the Bayer' arrangement.

Since the two arrangements described in Japanese Unexamined PatentApplication Publication No. 2000-316169 and EP Patent Publication No.0,804,037, (A2) employ a stronger random pattern than the Bayerarrangement or the pseudo-random Bayer arrangement, the occurrence of afalse color is reduced compared with the pseudo-random Bayer arrangementor the pseudo-random Bayer arrangement. However, since, in the twoarrangements, all the color filters are randomly arranged, theresolution is decreased compared with that of the Bayer arrangement orthe pseudo-random Bayer arrangement.

In general, in color separation processes, one color signal isinterpolated with a high resolution first. Thereafter, the other colorsignals are interpolated using that color signal as a reference.Accordingly, compared with the Bayer arrangement or the pseudo-randomBayer arrangement in which G color filters are arranged in a checkeredpattern and a correlation process described in Japanese Patent No.2931520 is used, it is very difficult for the arrangements described inJapanese Unexamined Patent Application Publication No. 2000-316169 andEP Patent Publication No. 0,804,037, (A2) to generate a reference color.Consequently, the reproducible frequency range is significantlydifferent for the position of each pixel.

Furthermore, since the arrangements described in Japanese UnexaminedPatent Application Publication No. 2000-316169 and EP Patent PublicationNo. 0,804,037, (A2) increase the number of colors compared with theBayer arrangement, the number of pixels for one color is reduced. Thisresults in a further decrease in resolution.

Still furthermore, in general, to read a signal out of a solid-stateimaging device at high speed, the signals from the pixels are thinnedout (dumped) or summed. The dumping and summing processes are cyclicallyexecuted. Accordingly, if a random filter arrangement is used, a filterpattern after dumping may be changed from the original pattern orsignals from different pixels may be summed.

As noted above, while a random pattern in the filter arrangement and theincrease in the number of colors reduce the occurrence of a false colorand increase the dynamic range and the performance of colorreproduction, the random pattern and the increase in the number ofcolors decrease the resolution and cause an unsuccessful operation ofdumping and summing the signals from the pixels.

SUMMARY OF THE INVENTION

As noted above, while the technology in which color filters used for asingle-plate color imaging device are randomly arranged in order toreduce the occurrence of a false color and the technology in which thenumber of colors used for filters are increased in order to improve theperformance of color reproduction and the dynamic range have beeninvented, the resolution is decreased compared with existing filterarrangements, such as the Bayer arrangement. In addition, a successfuloperation cannot be performed in the dumping readout method and thesumming readout method.

Accordingly, the present invention provides a color filter array, animaging device, and an image processing unit capable of sufficientlypreventing a decrease in resolution and the occurrence of a false colorand supporting dumping and summing processes of signals from pixels evenwhen filters of colors more than that of the Bayer arrangement are used.

According to an embodiment of the present invention, a color filterarray includes a plurality of filters, each having one of a plurality oftypes of spectral sensitivity and being disposed at the location of acorresponding one of a plurality of pixels. The filters of apredetermined type selected from among the plurality of types arearranged at the locations of the pixels in a checkered pattern, and thefilters of some or all of the other types are randomly arranged at thepixel locations at which the filters of the predetermined type are notpresent.

According to another embodiment of the present invention, a color filterarray includes a plurality of filters, each having one of at least fivetypes of spectral sensitivity and being disposed at the location of acorresponding one of a plurality of pixels. The filters of a first colorC1 a selected from among the at least five types of color are arrangedat the locations of the pixels on every other line in a horizontaldirection and a vertical direction, the filters of a second color C1 bselected from among the at least five types of color are arranged at thepixel locations at which the filters of the first color C1 a are notpresent on every other line in a horizontal direction and a verticaldirection, and the filters of some or all of the other colors arerandomly arranged at the pixel locations at which neither the filters ofthe first color C1 a nor the second color C1 b are present.

According to still another embodiment of the present invention, animaging device includes a color filter array including a plurality offilters, each having one of a plurality of types of spectral sensitivityand being disposed at the location of a corresponding one of a pluralityof pixels. The filters of a predetermined type selected from among theplurality of types are arranged at the locations of the pixels in acheckered pattern, and the filters of some or all of the other types arerandomly arranged at the pixel locations at which the filters of thepredetermined type are not present.

According to yet still another embodiment of the present invention, animaging device includes a color filter array including a plurality offilters, each having one of at least five types of spectral sensitivityand being disposed at the location of a corresponding one of a pluralityof pixels. The filters of a first color C1 a selected from among the atleast five types of color are arranged at the locations of the pixels onevery other line in a horizontal direction and a vertical direction, thefilters of a second color C1 b selected from among the at least fivetypes of color are arranged at the pixel locations at which the filtersof the first color C1 a are not present on every other line in ahorizontal direction and a vertical direction, and the filters of someor all of the other colors are randomly arranged at the pixel locationsat which neither the filters of the first color C1 a nor the secondcolor C1 b are present.

According to yet still another embodiment of the present invention, animage processing unit includes receiving means for receiving image datafrom an imaging device including a color filter array, firstinterpolating means, and second interpolating means. The color filterarray includes a plurality of filters, each having one of a plurality oftypes of spectral sensitivity and being disposed at the location of acorresponding one of a plurality of pixels. The filters of apredetermined type selected from among the plurality of types arearranged at the locations of the pixels in a checkered pattern, and thefilters of some or all of the other types are randomly arranged at thepixel locations at which the filters of the predetermined type are notpresent. The first interpolating means interpolates a pixel value of afirst color C1 at a pixel location of the image data received by thereceiving means at which the predetermined color is not present usingthe colors C1 present in the vicinity of the pixel location so as togenerate a first image. The second interpolating means interpolates apixel value of a second color CX different from the first color C1 usingthe first colors C1 and the second colors CX that are present in a localregion including a pixel of interest so as to generate a second image.

According to yet still another embodiment of the present invention, animage processing unit includes receiving means for receiving image datafrom an imaging device including a color filter array, firstinterpolating means, and second interpolating means. The color filterarray includes a plurality of filters, each having one of at least fivetypes of spectral sensitivity and being disposed at the location of acorresponding one of a plurality of pixels. The filters of a first colorC1 a selected from among the at least five types of color are arrangedat the locations of the pixels on every other line in a horizontaldirection and a vertical direction, the filters of a second color C1 bselected from among the at least five types of color are arranged at thepixel locations at which the filters of the first color C1 a are notpresent on every other line in a horizontal direction and a verticaldirection, and the filters of some or all of the other colors arerandomly arranged at the pixel locations at which neither the filters ofthe first color C1 a nor the second color C1 b are present. The firstinterpolating means interpolate a pixel value of a third color C1 c at apixel location of a pixel of interest using the pixel values of thefirst color C1 a and the second color C1 b that are present in a localregion including the pixel of interest in the image data received by thereceiving means. The second interpolating means interpolates a pixelvalue of each color CX of the plurality of filter colors including thecolor C1 a and the color C1 b by using the pixel values of the thirdcolor C1 c and the color CX that are present in the local regionincluding the pixel of interest.

According to the present invention, a color filter array, an imagingdevice, and an image processing unit capable of sufficiently preventinga decrease in resolution and the occurrence of a false color andsupporting dumping and summing processes of signals from a plurality ofpixels even when filters of colors more than that of the Bayerarrangement are used.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the structure of known color filterarray;

FIGS. 2A-2C illustrate the characteristics of examples of a color filterarray;

FIG. 3 is a diagram illustrating the structure of a color filter arrayaccording to a first exemplary embodiment of the present invention;

FIG. 4 is a diagram illustrating a modification of a color filter arrayaccording to the first exemplary embodiment of the present invention;

FIG. 5 illustrates an example in which signals are read out from asolid-state imaging device using a combination of the dumping readouttechnique and the summing readout technique;

FIG. 6 is a diagram in which pixels that are present in a local spaceare plotted in a G-B plane;

FIG. 7 is a diagram illustrating the structure of a color filter arrayhaving a five-color G checkered random arrangement according to thefirst exemplary embodiment of the present invention;

FIG. 8 is a block diagram of an exemplary structure of a digital videocamera according to an embodiment of the present invention;

FIG. 9 is a block diagram of processing of an image processing unitshown in FIG. 8 according to the embodiment of the present invention;

FIG. 10 is a flow chart of a process performed by a G interpolation unitshown in FIG. 9;

FIG. 11 is a flow chart of a process performed by an R interpolationunit shown in FIG. 9;

FIGS. 12A and 12B are flow charts of processes performed at step ST404and ST405 shown in FIG. 11, respectively;

FIG. 13 is a block diagram of a detailed structure of an R MS-SyncNRunit 207 shown in FIG. 9;

FIG. 14 is a block diagram illustrating an exemplary configuration of acorrection unit;

FIG. 15 illustrates an exemplary structure of a color filter arrayaccording to a second exemplary embodiment of the present invention;

FIG. 16 illustrates a demosaic process performed when the color filterarray shown in FIG. 15 is used;

FIG. 17 illustrates a modification of the color filter array accordingto the second exemplary embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A color filter array, an imaging device, and an image processing unitaccording to exemplary embodiments of the present invention are nowherein described.

First Exemplary Embodiment

According to a first exemplary embodiment, a filter arrangement is usedfor an imaging device in which one of a plurality of filters havingdifferent color separation characteristics (i.e., colors) is bonded toeach pixel. The filter arrangement includes four colors or more. A colorC1 is arranged in a checkered pattern. Some or all of the other colorsare randomly arranged at pixel locations at which the color C1 is notpresent.

As used herein, the term “color” refers to a filter or a pixel value ofthat color obtained from the filter.

In such a color arrangement, since color filters of four colors or moreare employed, the performance of color reproduction and the dynamicrange can be improved compared with a three-color filter arrangement.

Since the color C1 has a checkered pattern, a correlation process thatis applied to a G-color checkered pattern of the Bayer arrangement canbe applied to the C1 checkered pattern. Accordingly, a signal in ahigh-frequency range can be reproduced.

In addition, by using a random pattern, the problem of a false colorthat tends to occur for a color CX other than the color C1 having asmall number of pixels can be reduced. Furthermore, by estimating(interpolating) a high-frequency component of the color CX using thecolor C1 as a reference, the color CX that is finally obtained at everyposition of the pixel can be reproduced as a signal containing thehigh-frequency component.

An exemplary color filter arrangement according to the present exemplaryembodiment is shown in FIG. 3. This color filter arrangement is afive-color arrangement in which the color filters C1 are arranged in acheckered pattern and color filters C2, C3, C4, and C5 are randomlyarranged at positions at which the color filters C1 are not present.

In addition, for any pixel of interest, the occurrence frequency of acolor in an area including the pixel of interest and having apredetermined size is within a desired error range.

To estimate an unknown color at the position of a pixel of interest, thecolor needs to be present in the vicinity of the pixel of interest.

Accordingly, the occurrence frequency of a color in a local region needsto be within a desired range so as to avoid a filter arrangement inwhich only a specific color dominates.

An arrangement that satisfies the above-described conditions may beobtained through a plurality of attempts of random arrangements.Alternatively, as described in EP Patent Publication No. 0,804,037,(A2), a digital halftone technology in which colors are equally randomlydistributed in a local region may be applied.

As used herein, the term “randomly” does not necessarily mean“completely randomly”, but may mean “pseudo-randomly”. That is,repetition may occur over a long cycle.

Additionally, the occurrence frequency of a color in the end portions(the upper, lower, left, and light portions) of the filter arrangementmay be computed while taking into account a space in which the upper endis connected to the lower end of the arrangement and the left end isconnected to the right end of the arrangement. Alternatively, if thefilter arrangement is sufficiently large compared with a predeterminedarea size, the end portions may be negligible.

The color filter arrangement shown in FIG. 3 is a five-color filterarrangement in which the color filters C1 are arranged in a checkeredpattern and the color filters C2, C3, C4, and C5 are randomly arrangedat positions at which the color filters C1 are not present. The areahaving a predetermined size is a rectangle of 15 by 15 pixels. Theoccurrence frequency of the color filters C1 is determined as a ratio of1/2 whereas the occurrence frequency of each of the color filters C2,C3, C4, and C5 are determined as a ratio of 1/8. The error of theoccurrence frequency is determined to be ±1/50.

In addition, according to a first modification of the present exemplaryembodiment, a color filter arrangement is a four-color filterarrangement in which first color filters C1 are arranged in a checkeredpattern, second color filters C2 are arranged at positions at which thecolor filters C1 are not present on every other line in a horizontaldirection and in a vertical direction, and third color filters C3 andfourth color filters C4 are randomly arranged at positions at whichneither the color filters C1 nor C2 are present.

In such a filter arrangement, the color filters C1 and C2 are regularlyarranged whereas the color filters C3 and C4 are randomly arranged.

The color filters C1 correspond to a number of pixels about half thetotal number of pixels of the imaging device. The color filters C2correspond to a number of pixels about one fourth of the total number ofpixels of the imaging device.

The color filters C3 and C4 totally correspond to a number of pixelsonly one fourth of the total number of pixels of the imaging device.Accordingly, when the numbers of pixels having the color filters C3 issubstantially the same as the number of pixels having the color filtersC4, the number of the color filters C3 or C4 is about one eighth of thetotal number of pixels of the imaging device. Therefore, the resolutionis low and a false color easily occurs. However, by randomly arrangingthe color filters C3 and C4, the occurrence of a false color can bereduced.

In addition, in the above-described color filter arrangement accordingto the first modification, for example, the correlation between thespectral sensitivities of the color filters C3 and C4 is high.

In such a color filter arrangement, let the color filter C1 representthe G color, the color filters C2 represent the R color, the colorfilters C3 represent the B color, and the color filters C4 represent aB′ color having a spectral sensitivity close to that of the B color.Then, this color filter arrangement is considered to be an arrangementin which filters related to the B and B′ colors are randomly arranged atthe positions of filters related to the B color of the Bayerarrangement.

Hereinafter, this arrangement is referred to as a “four-color Gcheckered random arrangement”. The four-color G checkered randomarrangement is shown in FIG. 4.

Most techniques such as a dumping readout method or a summing readoutmethod for reading out signals from solid-state imaging devices areproposed on the basis of the Bayer arrangement. In these techniques, thearrangement obtained after the signals are read out is also the Bayerarrangement.

FIG. 5 illustrates an example in which signals are read out from asolid-state imaging device at high speed using a combination of thedumping readout method and the summing readout method.

In this example, four lines out of eight vertical lines are dumped. Inaddition, in the horizontal direction, the same color signals are summedwhile skipping every other line. In the vertical direction, the samecolor signals are summed while skipping every three lines.

When such a known readout method is applied to the four-color Gcheckered random arrangement, it is apparent that, according to thedumping readout method, a four-color G checkered random arrangement isgenerated after the readout operation is performed.

It is more desirable if the arrangement is determined in advance so thatthe above-described condition of the occurrence frequency is satisfiedeven after the dumping readout operation is performed.

In the case of the summing readout method, the same condition as theBayer arrangement is maintained for the G and R color signals. However,the B′ and B color signals are summed with no distinction and the summedsignal is output.

For example, when four pixels are summed, there are five cases asfollows:

(1) Four B-color pixels and zero B′-color pixels are summed;

(2) Three B-color pixels and one B′-color pixel are summed;

(3) Two B-color pixels and two B′-color pixels are summed;

(4) One B-color pixel and three B′-color pixels are summed; and

(5) Four B-color pixels and zero B′-color pixels are summed.

That is, even in a single-color region, pixel values obtained atpositions of the B color of the Bayer arrangement are differentdepending on the numbers of the B colors and the B′ colors.

However, let B″ denote a new color obtained by mixing the pixel valuesof the B color and the B′ color in proportion to the occurrencefrequency of color described referring to the color filter arrangementgenerated from the color separation computation of this embodiment.Then, the arrangement obtained by the summing readout method can beconsidered to be the Bayer arrangement of the R, G, and B″ colorfilters.

In the color separation process of the Bayer arrangement, if onlylow-frequency components of the R and B color signals are known, the Rand B color signals containing high-frequency components can beestimated using the G color signal as a reference.

For example, let G(p) denote the pixel value of the G color at a pixellocation p, and let G_(low) (p) and B_(low) (p) respectively denote thelow-frequency components of the G and B color signals. Then, a pixelvalue B(p) of the B color can be estimated using the following equation:

$\begin{matrix}{{B(p)} = {\frac{B_{low}(p)}{G_{low}(p)}{G(p)}}} & (1)\end{matrix}$

The low-frequency component is obtained by averaging the G color signalsor the B color signals from pixels that are present in the vicinity ofthe pixel location p. In the greater part of an RGB image, the pixelsvalues of the R, G, and B colors have a strong positive correlation.

Accordingly, as shown in FIG. 6, when the pixels that are present in alocal region are plotted in a space where the abscissa represents thepixel value of the G color and the ordinate represents the pixel valueof the B color, the pixels are distributed within a limited smallregion.

This distribution is approximated by a linear regression line passingthrough the origin and the center point of the distribution, that is, bya line indicated by equation (1).

Here, a computation method used for obtaining a low-frequency componentof the B color signal in the Bayer arrangement is applied to thearrangement obtained by the summing readout method.

In this computation method, the sum of the B and B′ color signals arefurther averaged in a given local region.

When considered from a computation for an arrangement before summing isperformed, this computation only involves averaging of the B and B′color signals from the pixels that are present in the vicinity of thepixel location p.

Since the spectral sensitivity of the B color filter is close to that ofthe B′ color filter, it is expected that the both color filters havesubstantially the same frequency characteristic for the same incidentlight pattern. Furthermore, if the local region is sufficiently large,it is expected that the ratio of the number of pixels of the summed Bcolor to that of the summed B′ color is close to the above-describedoccurrence frequency.

According to this feature, the low-frequency component of the B″ colorsignal in the Bayer arrangement of the R, B, and B″ color filters can beapproximated using the arrangement obtained by the summing readoutmethod.

If the low-frequency component of the B″ color signal is obtained, thepixel value of the B″ color can be estimated using the G color as areference, as indicated by equation (1). Accordingly, the arrangementobtained by the summing readout method can be used in place of the Bayerarrangement of the R, G, and B″ color filters.

Similarly, in the case where the dumping readout method is combined withthe summing readout method, the obtained arrangement can be used inplace of the Bayer arrangement of the R, G, and B″ color filters.

According to a second modification of the present exemplary embodiment,a color filter arrangement is a five-color filter arrangement in whichfirst color filters C1 are arranged in a checkered pattern, second colorfilters C2 and third color filters C3 are randomly arranged at positionsat which the color filters C1 are not present on every other line in ahorizontal direction and in a vertical direction, and fourth colorfilters C4 and fifth color filters C5 are randomly arranged at positionsat which neither the color filters C1 nor C2 nor C3 are present.

In such a filter arrangement, by increasing the number of colors of thefilter arrangement, the performance of color reproduction and thedynamic range can be further increased.

In the color filter arrangement according to the second modification,for example, the correlation of spectral sensitivity between the colorfilters C2 and C3 may be set to be high, and the correlation of spectralsensitivity between the color filters C4 and C5 may be set to be high.

In such a color filter arrangement, for example, let the color filter C1represent the G color, the color filter C2 represent the R color, thecolor filter C3 represent the R′ color having the sensitivity close tothat of the R color filter, the color C4 represent the B color, and thecolor filter C5 represent a B′ color having the spectral sensitivityclose to that of the B color filter. Then, this color filter arrangementis considered to be an arrangement in which the R and R′ color filtersare randomly arranged at the positions of the R color filters of theBayer arrangement, and the B and B′ color filters are randomly arrangedat the positions of the B color filter of the Bayer arrangement.

Hereinafter, this arrangement is referred to as a “five-color Gcheckered random arrangement”. The five-color G checkered randomarrangement is shown in FIG. 7.

In the case where the dumping readout method or the summing readoutmethod is applied to the five-color G checkered random arrangement, letR″ denote a new color obtained by mixing the pixel values of the R colorand the R′ color in proportion to the occurrence frequency of a colorfilter described referring to the color filter arrangement of thisembodiment, and let B″ denote a new color obtained by mixing the pixelvalues of the B color and the B′ color in proportion to the occurrencefrequency of a color filter described referring to the color filterarrangement of this embodiment. Then, the arrangement obtained afterreading out the signals can be considered to be the Bayer arrangement ofthe R″, G, and B″ color filters.

An image capturing apparatus (a digital video camera) using the colorfilter arrangement according to the above-described embodiment isdescribed next.

FIG. 8 is a block diagram of an exemplary structure of a digital videocamera 100 according to an embodiment of the present invention.

As shown in FIG. 8, the digital video camera 100 includes a lens 101, anaperture 102, a charge-coupled device (CCD) image sensor 103, acorrelated double sampling circuit 104, an A/D converter 105, a digitalsignal processor (DSP) block 106, a timing generator 107, a D/Aconverter 108, a video encoder 109, a video monitor 110, a coder decoder(CODEC) 111, a memory 112, a central processing unit (CPU) 113, and aninput device 114.

Here, the input device 114 includes operation buttons, such as arecording button mounted on the body of the digital video camera 100.

The DSP block 106 is a block including a signal processing processor andan image RAM (an image memory). The signal processing processor canperform a pre-programmed process on image data stored in the RAM.Hereinafter, the DSP block is simply referred to as a “DSP”.

Incident light arrived at the CCD image sensor 103 through an opticalsystem is received by each of light receiving elements on the imageplane of the CCD image sensor 103. The light receiving elementsphotoelectrically convert the incident light to an electrical signal.The correlated double sampling circuit 104 removes noise in theelectrical signal. The A/D converter 105 digitizes the electricalsignal. Thereafter, the DSP 106 temporarily stores the signal in theimage memory.

During capturing an image, the timing generator 107 controls a signalprocessing system to capture the image at a constant frame rate. A pixelstream is transferred to the DSP 106 at a constant rate. The DSP 106performs appropriate image processing on the pixel stream and deliversimage data to the D/A converter 108 or the CODEC 111 or the both.

The D/A converter 108 converts the image data delivered from the DSP 106to an analog signal. Thereafter, the video encoder 109 converts theanalog signal to a video signal. A user can monitor the video signalthrough the video monitor 110. The video monitor 110 serves as a finderof the camera according to the present exemplary embodiment.

The CODEC 111 encodes the image data delivered from the DSP 106 andstores the encoded image data in the memory 112. The memory 112 may bereplaced with a recording unit using a semiconductor, a magneticrecording medium, a magneto optical recording medium, or an opticalrecording medium.

The digital video camera according to the present exemplary embodimentincludes such components. The feature of the above-described embodimentis applied to image processing performed by the DSP 106. This imageprocessing is described in detail next.

As noted above, according to the present exemplary embodiment, an imageprocessing unit is realized by using the DSP 106. Accordingly, in theconfiguration of the present exemplary embodiment, the operation of theimage processing unit is realized by an arithmetic unit in the DSP 106that sequentially performs computation described in predeterminedprogram code on the input stream of the image signal.

In the following description, each of the processing modules of theprogram code is described as a functional block, and the sequence ofperforming the modules is described using a flow chart. However, inaddition to a program described below, the present invention can berealized by a hardware circuit that performs processes that are the sameas the following processes.

Here, an on-chip color filter array of the CCD image sensor 103 employsthe filter arrangement according to the above-described embodiment ofthe present invention. In the temporarily stored image, each pixel hasonly one color. The DSP 106 processes this image in accordance with aprestored image processing program so as to generate image data in whicheach pixel has a full color.

According to the present exemplary embodiment, description is made withreference to image processing on a mosaic image obtained from thefive-color G checkered random arrangement shown in FIG. 7.

The G pixel values at all of pixel locations are computed using the Gpixel values obtained in a checkered pattern.

Subsequently, using the G pixel value as a reference, R, R′, B, and B′pixel values are interpolated for all the pixel locations.

Thereafter, the MS-SyncNR is applied to an image in which pixel valuesof all the colors are allocated to all the pixel locations so that theoccurrence of a false color is eliminated.

FIG. 9 is a block diagram of the image processing unit according to thepresent exemplary embodiment. The RR′GBB′ mosaic image obtained from thefive-color G checkered random arrangement is input to the imageprocessing unit. A G interpolation unit 201, an R interpolation unit202, an R′ interpolation unit 203, a B interpolation unit 204, and a B′interpolation unit 205 generate a first RR′GBB′ interpolated image 206which is an image in which pixel values of all the color areinterpolated at all the pixel locations. In addition, an R MS-SyncNRunit 207, an R′ MS-SyncNR unit 208, a B MS-SyncNR unit 209, and a B′MS-SyncNR unit 210 process this first RR′GBB′ interpolated image 206 soas to generate a second RR′GBB′ interpolated image 211 as a finaloutput.

G Interpolation Unit

In the G interpolation unit 201, the G pixel value is interpolated atall the pixel locations.

Since the number of G color pixels is large compared with the othercolor pixels, a high-resolution interpolated image can be obtained evenwhen a simple interpolation method, such as the Bicubic method, isapplied.

Here, description is made using the method described in Japanese PatentNo. 2931520 entitled “Color Separation Circuit of Single-plate ColorVideo Camera”, which is one of interpolation methods effective for thearrangement having a checkered G color pattern.

FIG. 10 is a flow chart of a process performed by the G interpolationunit 201 shown in FIG. 9.

Hereinafter, description is made with reference to this flow chart.

In a loop 302, the G interpolation unit 201 repeatedly performs aprocess for each of the pixel locations. As used herein, a pixel that issubjected to the process in one loop is referred to as a “pixel ofinterest”.

At step ST301, the G interpolation unit 201 reads out the pixel valuesof pixels in the vicinity of a pixel of interest in the mosaic image.

Subsequently, at step ST303, the G interpolation unit 201 determineswhether the filter color at the location of the pixel of interest is G.

If, at step ST303, the G interpolation unit 201 determines that thefilter color at the location of the pixel of interest is G, theprocessing proceeds to step ST304, where the pixel value of the pixel ofinterest is considered to be a G color pixel value.

However, if, at step ST303, the G interpolation unit 201 determines thatthe filter color at the location of the pixel of interest is not G, theprocessing proceeds to step ST305.

At step ST305, the G interpolation unit 201 computes a horizontalgradient GradH using the following equation:

GradH=|M(x−1,y)−M(x+1,y)|  (2)

where M(x, y) represents the pixel value at a location (x, y) of thepixel of interest.

Subsequently, the G interpolation unit 201 computes a horizontalinterpolation GH using the following equation:

$\begin{matrix}{{GH} = \frac{{M\left( {{x - 1},y} \right)} + {M\left( {{x + 1},y} \right)}}{2}} & (3)\end{matrix}$

In the same manner, at step ST306, the G interpolation unit 201 computesa vertical gradient GradV using the following equation:

GradH=|M(x,y−1)−M(x,y+1)|  (4)

Subsequently, the G interpolation unit 201 computes a verticalinterpolation GV using the following equation:

$\begin{matrix}{{GV} = \frac{{M\left( {x,{y - 1}} \right)} + {M\left( {x,{y + 1}} \right)}}{2}} & (5)\end{matrix}$

At step ST307, the G interpolation unit 201 interpolates a pixel valueof the G color G(x, y) of the pixel of interest using GradH, GradV, GH,and GV according to the following equation:

$\begin{matrix}{{{G\left( {x,y} \right)} = {{\frac{GradV}{{GradH} + {GradV}}{GH}} + {\frac{GradH}{{GradH} + {GradV}}{GV}}}}{{G\left( {x,y} \right)} = {{\frac{GradV}{{GradH} + {GradV}}{GH}} + {\frac{GradH}{{GradH} + {GradV}}{GV}}}}} & (6)\end{matrix}$

The interpolation process of the G pixel value of the pixel of interestis completed when the process at step ST304 or ST307 is completed.Thereafter, the next loop process for the location of the next pixel ofinterest starts. If the loop process for all the pixels is completed,the processing exits the loop 302. Thus, the process performed by the Ginterpolation unit 201 is completed.

R Interpolation Unit 202, R′ Interpolation Unit 203, B InterpolationUnit 204, and B′ Interpolation Unit 205

The processes performed by the R interpolation unit 202, the R′interpolation unit 203, the B interpolation unit 204, and the B′interpolation unit 205 are similar except regarding the target color.Accordingly, description is made with reference to only the Rinterpolation unit 202. Descriptions of the R′ interpolation unit 203,the B interpolation unit 204, and the B′ interpolation unit 205 can beobtained by replacing the symbol “R” in the description of the Rinterpolation unit 202 with “R′”, “B”, and “B′”, respectively.

FIG. 11 is a flow chart of a process performed by the R interpolationunit 202 shown in FIG. 9. Hereinbelow, a procedure is described withreference to this flow chart.

In a loop 403, the R interpolation unit 202 repeatedly performs aprocess for each of the pixel locations. As used herein, a pixel that issubjected to the process in one loop is referred to as a “pixel ofinterest”.

At step ST401, the R interpolation unit 202 reads out the G pixel valuesof the pixels in the vicinity of a pixel of interest. Note that thepixel values were computed by the G interpolation unit 201 shown in FIG.9.

In addition, at step ST402, the R interpolation unit 202 reads out thepixel values of pixels in the vicinity of the pixel of interest in theRR′GBB′ mosaic image.

Subsequently, at step ST404, the R interpolation unit 202 computes alow-frequency component G_(low) of the G color pixel, which is the pixelof interest. At step ST405, the R interpolation unit 202 computes alow-frequency component R_(low) of the R color pixel, which is the pixelof interest.

At step ST406, the R interpolation unit 202 interpolates an R pixelvalue R(x, y) of the pixel of interest using G_(low) and R_(low)according to the following equation:

$\begin{matrix}{{R\left( {x,y} \right)} = {\frac{R_{low}}{G_{low}}{G\left( {x,y} \right)}}} & (7)\end{matrix}$

The interpolation process of the R pixel value of the pixel of interestis completed when the process at step ST406 is completed. Thereafter,the next loop process for the location of the next pixel of intereststarts. If the loop process for all the pixels is completed, theprocessing exits the loop 403. Thus, the process performed by the Rinterpolation unit 202 is completed.

FIG. 12A is a flow chart of a detailed process performed at step ST404shown in FIG. 11.

In this process, the G color pixels contained in a local region areaveraged. That is, a low-pass filter using a finite impulse response(FIR) filter is formed.

At step ST501, the R interpolation unit 202 initializes G_(low) to zero.

Subsequently, at step ST502, the R interpolation unit 202 performs aloop process for each of the locations of all the pixels in a localregion including the pixel of interest.

As used herein, the term “local region” refers to the above-describedregion having the predetermined size.

Additionally, a pixel that is subjected to the process in one loop isreferred to as a “pixel of interest in the local region”.

At step ST503, the R interpolation unit 202 multiplies G(s, t) by WG(s,t). Thereafter, the resultant value is added to G_(low). The resultantvalue is considered to be a new value of G_(low).

Here, G(s, t) denotes the pixel value of G color at the location (s, t)of a pixel of interest in the local region. WG(s, t) denotes a weightingcoefficient.

WG(s, t) is determined to be a coefficient for a low-pass filter and thesum of the coefficients is 1.

The process for the pixel of interest in the local region is completedwhen the process at step ST503 is completed. Thereafter, the next loopprocess for the next pixel of interest in the local region starts. Ifthe loop process for all the pixels in the local region is completed,the processing exits the loop 502. Thus, the process performed by the Rinterpolation unit 202 at step ST404 is completed.

FIG. 12B is a flow chart of a detailed process performed at step ST405shown in FIG. 11.

In this process, the R color pixels contained in the local region areaveraged. That is, a low-pass filter using an FIR filter is formed.

At step ST601, the R interpolation unit 202 initializes a variableR_(low) to 0.

Thereafter, in a loop 602, the R interpolation unit 202 performs a loopprocess for each of the locations of all the pixels in a local regionincluding the pixel of interest.

Here, the local region refers to the above-described region having thepredetermined size.

At step ST603, the R interpolation unit 202 determines whether thefilter color at the location of the pixel of interest in the localregion is R.

If, at step ST603, the R interpolation unit 202 determines that thefilter color at the location of the pixel of interest is R, theprocessing proceeds to step ST604, where M(s, t) is multiplied by WR(s,t). The resultant value is added to R_(low). Then, the resultant valueis considered to be a new value of R_(low).

Here, M(s, t) denotes the pixel value at the location (s, t) of a pixelof interest in the local region. WR(s, t) denotes a weightingcoefficient.

WR(s, t) is determined to be a coefficient for a low-pass filter and thesum of the coefficients is 1.

However, since the R color pixels are randomly arranged, a differentWR(s, t) is used in accordance with the position of the R color pixelsin the local region.

In addition, it is desirable that WR(s, t) is determined so that thecharacteristics of the low-pass filter composed of WR(s, t) are close tothose of a low-pass filter composed of WG(s, t).

The process for this pixel of interest in the local region is completedwhen the process at step ST604 is completed. Thereafter, the next loopprocess for the next pixel of interest in the local region starts. Ifthe loop process for all the pixels in the local region is completed,the processing exits the loop 602. Thus, the process performed by the Rinterpolation unit 202 at step ST405 is completed.

R MS-SyncNR Unit 207, R′ MS-SyncNR Unit 208, B MS-SyncNR Unit 209, andB′ MS-SyncNR Unit 210

The processes performed by the R MS-SyncNR unit 207, the R′ MS-SyncNRunit 208, the B MS-SyncNR unit 209, and the B′ MS-SyncNR unit 210 aresimilar except regarding the target color. Accordingly, description ismade with reference to only the R MS-SyncNR unit 207. Descriptions ofthe R′ MS-SyncNR unit 208, the B MS-SyncNR unit 209, and the B′MS-SyncNR unit 210 can be obtained by replacing the symbol “R” in thedescription of the R MS-SyncNR unit 207 with “R′”, “B”, and “B′”,respectively.

FIG. 13 is a block diagram of a detailed structure of the R MS-SyncNRunit 207 shown in FIG. 9.

In the structure shown in FIG. 13, two G and R channels receive signals,and noise is removed from the R color signal. Thereafter, an outputsignal without noise is output from the structure. If a component thathas no correlation with the R color signal is removed from the G colorsignal, the occurrence of a false color is eliminated. Accordingly, inthe present exemplary embodiment, noise is not removed from the G colorsignal. However, in practical applications, it is desirable that noiseis removed from the G color signal using some noise removing method and,subsequently, noise is removed from the R color signal by the methodaccording to the present exemplary embodiment.

The R MS-SyncNR unit 207 includes two multiple-resolution transformunits 701 and 702, a multiple-resolution inverse transform unit 717, andcorrection units 711, 712, and 713. Note that the number of correctionunits (three in FIG. 13) is determined by subtracting one from thenumber of layers of the multiple resolutions. The firstmultiple-resolution transform unit 701 converts an image input from theG channel to multiple-resolution image data. Thereafter, themultiple-resolution transform unit 701 stores image signalscorresponding to the layers of the multiple resolutions in memories 703,704, 705, and 706.

Similarly, the second multiple-resolution transform unit 702 converts animage input from the R channel to multiple-resolution image data.Thereafter, the multiple-resolution transform unit 702 stores imagesignals corresponding to the layers of the multiple resolutions inmemories 707, 708, 709, and 710.

The correction units 711, 712, and 713 correspond to layers other thanthe layer of the minimum resolution. Each of the correction units 711,712, and 713 receives images of the G and R channels in thecorresponding layer, corrects each of the pixels of the images thatcontain noise so as to generate an R channel image without noise, andstores the generated R channel image in the corresponding one of imagememories 714, 715, and 716.

The correction units 711, 712, and 713 are described in more detailnext.

Since the correction units 711, 712, and 713 have a similarconfiguration and operate in a similar manner, only the correction unit711 is described here.

FIG. 14 is a block diagram illustrating an exemplary configuration ofthe correction unit 711.

As shown in FIG. 14, the correction unit 711 includes samplingprocessing units 801 and 802, a variance computing unit 803, a clippingprocessing unit 804, a divider processing unit 805, a covariancecomputing unit 806, and a multiplier processing unit 807.

The sampling processing unit 801 samples (extracts) a plurality of Gchannel pixel values from pixels in the vicinity of a predeterminedlocation determined corresponding to the location of a pixel ofinterest. The sampling processing unit 801 then delivers the sampled Gchannel pixel values to the multiplier processing unit 807, the variancecomputing unit 803, and the covariance computing unit 806. The samplingprocessing unit 802 samples (extracts) a plurality of R channel pixelvalues from pixels in the vicinity of a predetermined locationdetermined corresponding to the location of a pixel of interest. Thesampling processing unit 802 then delivers the sampled R channel pixelvalues to the covariance computing unit 806. Note that the samplingprocessing units 801 and 802 extract the G pixel value and the R pixelvalue at the same location determined corresponding to the position ofthe pixel of interest.

The variance computing unit 803 computes the variance of the G pixelvalues in the vicinity of the pixel of interest on the basis of thesampled G pixel values. Thereafter, the variance computing unit 803delivers the computed variance to the clipping processing unit 804.

The covariance computing unit 806 computes the covariance of the G pixelvalues and the R pixel values on the basis of the sampled G and R pixelsvalues. The covariance computing unit 806 then delivers the computedcovariance to the divider processing unit 805. If the variance of thesamples of the G channel is less than a predetermined threshold value,the clipping processing unit 804 clips the variance to the predeterminedthreshold value and delivers that value to the divider processing unit805. The divider processing unit 805 divides the covariance deliveredfrom the covariance computing unit 806 by the variance delivered fromthe clipping processing unit 804. Thus, the divider processing unit 805delivers the ratio of the covariance to the variance (i.e., thecovariance/the variance) to the multiplier processing unit 807. Theabove-described process performed by the clipping processing unit 804prevents the occurrence of the ratio of zero when the divider processingunit 805 performs the subsequent process (i.e., the computation of thecovariance/the variance). By multiplying the value (the computation ofthe covariance/the variance) by the G channel pixel value at thelocation of the pixel of interest, the multiplier processing unit 807estimates the R channel pixel value of the pixel of interest withoutnoise and outputs the estimated value.

The operations of the correction units 711, 712, and 713 are describedin detail next.

The correction units 711, 712, and 713 correct pixel values by a pixelvalue estimation method using a correlation between channels. Morespecifically, on ground that there is a linear relationship between twochannels (e.g., G and R) when focusing on a local region, the estimationvalue of R at a pixel location in the local region is obtained by alinear regression computation.

For example, when samples {C₁₁, C₁₂, C₁₃, . . . C_(1N)} of the pixelvalues of a C₁ channel (e.g., a G color channel) and samples {C₂₁, C₂₂,C₂₃, . . . C_(2N)} of the pixel values of a C₂ channel (e.g., an R colorchannel) are acquired in the local region around the pixel of interestin an image and there is a linear relationship between the two channel,a luminance estimation value C_(2C)′ of the C₂ channel at the locationof the pixel of interest can be obtained using a luminance estimationvalue C_(1C) of the C₁ channel at the location of the pixel of interestaccording to the following equation:

$\begin{matrix}{C_{2c} = {{\frac{V_{C_{1}C_{2}}}{V_{C_{1}C_{1}}}\left( {C_{1c} - M_{C_{1}}} \right)} + M_{C_{2}}}} & (8)\end{matrix}$

where M_(C1) denotes the expectation value of the C₁ channel in thelocal region, M_(C2) denotes the expectation value of the C₂ channel,V_(C1C2) is the covariance of the C₁ and C₂ channels, and V_(C1C1) isthe variance of the C₁ channel. Note that the samples are the pixelvalues of a plurality of pixels at predetermined locations correspondingto the location of the pixel of interest.

In addition, the covariance V_(C1C2) and the variance V_(C1C1) can beobtained using the following equations (9) and (10):

$\begin{matrix}{V_{C_{1}C_{2}} = \frac{\sum\limits_{i = 1}^{N}\left\lbrack {{w_{i} \cdot \left( {C_{1_{i}} - M_{C_{1}}} \right)}\left( {C_{2_{i}} - M_{C_{2}}} \right)} \right\rbrack}{\sum\limits_{i = 1}^{N}w_{i}}} & (9) \\{V_{C_{1}C_{2}} = \frac{\sum\limits_{i = 1}^{N}\left\lbrack {w_{i} \cdot \left( {C_{1_{i}} - M_{C_{1}}} \right)^{2}} \right\rbrack}{\sum\limits_{i = 1}^{N}w_{i}}} & (10)\end{matrix}$

In equations (9) and (10), w_(i) denotes a predetermined weightingcoefficient.

Accordingly, by solving equation (8), noise can be eliminated. However,as shown in FIG. 13, the R MS-SyncNR unit 207 includes the resolutiontransform units 701 and 702. Therefore, the correction unit 711 correctsthe pixel values for images separated for each of frequency ranges usingthe above-described correlation between two channels.

In addition, among images in a plurality of layers generated by themultiple resolution conversion, direct-current components areconcentrated in the minimum-resolution layer. Accordingly, theexpectation values of local pixels in the layers other than theminimum-resolution layer are zero. As a result, when the multipleresolution process is used, equations (8) to (10) are replaced by thefollowing equations (11) to (13):

$\begin{matrix}{C_{2c} = {\frac{V_{C_{1}C_{2}}}{V_{C_{1}C_{1}}}C_{1c}}} & (11) \\{V_{C_{1}C_{2}} = \frac{\sum\limits_{i = 1}^{N}\left\lbrack {w_{i} \cdot C_{1_{i}} \cdot C_{2_{i}}} \right\rbrack}{\sum\limits_{i = 1}^{N}w_{i}}} & (12) \\{V_{C_{1}C_{1}} = \frac{\sum\limits_{i = 1}^{N}\left\lbrack {w_{i} \cdot C_{1_{i}}^{2}} \right\rbrack}{\sum\limits_{i = 1}^{N}w_{i}}} & (13)\end{matrix}$

Accordingly, in practice, the variance computing unit 803 shown in FIG.14 computes the variance using equation (13). The covariance computingunit 806 computes the covariance using equation (12). Additionally, themultiplier processing unit 807 modifies (corrects) the pixel valuesusing equation (11).

Furthermore, to reduce the high processing load of the multiplicationprocess performed by a computer when the covariance and the variance arecomputed using equations (12) and (13), an approximate function may beused. Thus, the number of the multiplication processing operations maybe reduced. For example, the following approximate functions may beused:

$\begin{matrix}{{V_{C_{1}C_{2}} = \frac{\sum\limits_{i = 1}^{N}\left\lbrack {2{w_{i} \cdot {{sgn}\left( {C_{1_{i}},C_{2_{i}}} \right)} \cdot {\min \left( {{C_{1_{i}}},{C_{2_{i}}}} \right)}}} \right\rbrack}{\sum\limits_{i = 1}^{N}w_{i}}}{{{sgn}\left( {a,b} \right)}\left\{ \begin{matrix}1 & {\left( {\left( {a > 0} \right)\bigwedge\left( {b > 0} \right)} \right)\bigvee\left( {\left( {a < 0} \right)\bigwedge\left( {b < 0} \right)} \right)} \\0 & {\left( {a = 0} \right)\bigvee\left( {b = 0} \right)} \\{- 1} & \left( {{\left( {a > 0} \right)\bigwedge\left( {b < 0} \right)}\bigvee\left( {\left( {a > 0} \right)\bigwedge\left( {b < 0} \right)} \right)} \right.\end{matrix} \right.}} & (14) \\{V_{C_{1}C_{1}} = \frac{\sum\limits_{i = 1}^{N}\left\lbrack {2{w_{i} \cdot {C_{1_{i}}}}} \right\rbrack}{\sum\limits_{i = 1}^{N}w_{i}}} & (15)\end{matrix}$

In addition, the multiple resolution inverse transform unit 717 receivesthe R channel images without noise in the layers and the R channel imagein the minimum-resolution layer and combines all the R channel imagesinto an image having the same resolution as that of the original image.Subsequently, the multiple resolution inverse transform unit 717 outputsthe combined image.

As noted above, according to the present exemplary embodiment, a colorfilter arrangement used for a single-plate color imaging device isdefined in which filters of four or more color are included, the filtersof some of the colors are regularly arranged, and filters of the othercolors are randomly arranged. Then, an image processing unit includes acolor filter array having such an arrangement.

According to the present exemplary embodiment, the color filter arrayhaving such an arrangement uses the number of colors more than that usedin the Bayer arrangement, and therefore, can improve the performance ofthe color reproduction and the dynamic range.

In addition, since the color filter array includes regularly arrangedcolor filters, the color filter array can provide the resolution that isthe same as that of the Bayer arrangement. Additionally, by using arandom filter arrangement and the R MS-SyncNR unit 207, the R′ MS-SyncNRunit 208, the B MS-SyncNR unit 209, and the B′ MS-SyncNR unit 210 shownin FIG. 9, this color filter array can efficiently reduce the occurrenceof a false color.

That is, according to the present exemplary embodiment, since the C1color occupies a half of the total number of pixels in a mosaic imageobtained through the color filter array, the C1 color has a resolutionhigher than that of the other color. Accordingly, the C1 color isallocated to the positions of all the pixels first. Subsequently,high-frequency components of the other colors are estimated(interpolated) using the allocated C1 color as a reference. In this way,an image with high resolution, for all the colors can be obtained afterthe interpolation is performed.

In addition, according to the present exemplary embodiment, in the imageobtained by image-processing the mosaic image obtained through the colorfilter array, a false color that remarkably occurs in a specific spatialfrequency range is prevented. On the contrary, a few false colors aredistributed in a variety of spatial frequency ranges. Consequently, thefalse colors are not readily perceived by the human eye. In addition, byusing the R MS-SyncNR unit 207, the R′ MS-SyncNR unit 208, the BMS-SyncNR unit 209, and the B′ MS-SyncNR unit 210, these false colorscan be efficiently eliminated.

In the process using the R MS-SyncNR unit 207, the R′ MS-SyncNR unit208, the B MS-SyncNR unit 209, and the B′ MS-SyncNR unit 210, areference channel (e.g., G) signal is determined and estimation isperformed so that the correlation of the other channel signal withrespect to the reference signal is the highest on the basis of the sameidea as the pixel value estimation method using a correlation betweenchannels. Thus, components having no correlation between channels arereduced and chrominance non-uniformity and a color registration errorcaused by noise components mixed in the chrominance components can beeliminated. According to the present exemplary embodiment, the obtainedimage has the same characteristic as an image with chrominancenon-uniformity. A low correlation between channels results in an imagehaving false colors distributed in a variety of spatial frequencyranges. Therefore, by increasing the correlation between channels usingthe R MS-SyncNR unit 207, the R′ MS-SyncNR unit 208, the B MS-SyncNRunit 209, and the B′ MS-SyncNR unit 210, the occurrence of the falsecolor can be eliminated.

According to the present exemplary embodiment, the arrangement of thecolor filter array can be considered to be the same as the Bayerarrangement when a dumping readout process or a summing readout processis performed.

Second Exemplary Embodiment

According to a second exemplary embodiment, a color filter arrangementis obtained by replacing the C1 color filters arranged in a checkeredpattern in the color filter arrangement according to the first exemplaryembodiment with regularly arranged filters of two colors C1 a and C1 b.

As described in Japanese Unexamined Patent Application Publication No.2005-160044, a new color C1 c is generated on the basis of the colors C1a and C1 b. Thereafter, using the regular arrangement of the C1 a and C1b color filters, the color C1 c can be interpolated at the locations ofall of the pixels at which the C1 a or C1 b color filters are present.

The filter arrangement in which the C1 c color filters are arranged inplace of the C1 a and C1 b color filters is the same as the filterarrangement described in the first exemplary embodiment.

That is, according to the second exemplary embodiment, the color filterarray provides the same advantage as that of the first exemplaryembodiment. In addition, since the number of filter colors isincremented by one compared with the first exemplary embodiment, theperformance of color reproduction and the dynamic range can be furtherimproved.

FIG. 15 illustrates an exemplary structure of the color filter arrayaccording to the second exemplary embodiment.

As shown in FIG. 15, in the color filter array according to the presentexemplary embodiment, the filters of two colors G and G′ are arranged ina checkered pattern and the filters of other four colors are arranged ina random pattern.

FIG. 16 illustrates a demosaic process performed when the color filterarray shown in FIG. 15 is used.

When compared with the image processing unit shown in FIG. 9, as shownin FIG. 16, the image processing unit according to the present exemplaryembodiment includes a (G+G′) interpolation unit a201 in place of the Ginterpolation unit 201. Additionally, processes similar to thoseperformed by the R interpolation unit 202, the R′ interpolation unit203, the B interpolation unit 204, and the B′ interpolation unit 205 areadded to the processes for the G and G′ colors (i.e., a G interpolationunit a202 and a G′ interpolation unit a203). Thus, a six-plane image(i.e., a first RR′GG′BB′ interpolation image a208) is generated insteadof a five-plane image.

Since noise caused by a random arrangement does not occur in the G colorimage and the G′ color image, the subsequent noise removing process isskipped. Thus, like the process shown in FIG. 9, the MS-SyncNR processis performed on each of the R, R′, B, and B′ color images.

To realize the (G+G′) interpolation unit a201, the interpolationalgorithm described in Japanese Unexamined Patent ApplicationPublication No. 2005-160044 can be applied. Japanese Unexamined PatentApplication Publication No. 2005-160044 describes an interpolationprocess in which, from an arrangement in which filters of two colors C1and C2 are arranged in a checkered pattern, a color C3 (=C1+C2) isinterpolated for all of a plurality of pixels.

In addition, the block structure shown in FIG. 8 and the flow chartshown in FIG. 9 described in the first exemplary embodiment can beapplied to the present exemplary embodiment.

FIG. 17 illustrates a modification of the color filter array accordingto the present exemplary embodiment. In this color filter array, filtersof two colors G and G′ are arranged in a checkered pattern, filters of acolor R are arranged on every other pixel in the horizontal directionand the vertical direction, and filters of the other two colors arerandomly arranged.

According to the present exemplary embodiment, by replacing, forexample, the color filters C1 with the color filters C1 a and C1 b, theperformance of color reproduction and the dynamic range can be furtherimproved.

In addition, it is desirable that the correlation of spectralsensitivity between the colors C1 a and C1 b is higher than that betweenthe other colors. That is, as described in Japanese Unexamined PatentApplication Publication No. 2005-160044, when a new color C1 c isgenerated on the basis of the C1 a and C1 b colors using the regulararrangement of the C1 a and C1 b color filters, it is desirable that thecorrelation of spectral sensitivity between the C1 a and C1 b colorfilters is high.

In addition, in a process in which a color image having all pixels of afull-color is generated from a mosaic image obtained by an imagingdevice including the color filter array according to the presentexemplary embodiment, a new color C1 c is computed using the colors C1 aand C1 b.

If the color filter C1 a is located at the position of the pixel ofinterest, the image processing unit estimates (interpolates) the pixelvalue of the C1 b color at the position of the pixel of interest usingthe C1 a and C1 b colors that are present in the local region includingthe position of the pixel of interest.

Moreover, if the color filter C1 b is located at the position of thepixel of interest, the image processing unit interpolates the pixelvalue of the C1 a color at the position of the pixel of interest usingthe C1 a and C1 b colors that are present in the local region includingthe position of the pixel of interest.

Furthermore, the image processing unit computes the pixel values of thecolor C1 c at the locations of pixels arranged in a checkered patternfrom the pixel values of the colors C1 a and C1 b at those locations.The image processing unit interpolates the pixel values of the color C1c at pixel locations at which the color C1 c is not present using thepixel values of the color C1 c present around the pixel locations. Inaddition, the image processing unit interpolates a color CX other thanthe color C1 c using the pixel values of the colors C1 c and CX that arepresent in the local region including the position of the pixel ofinterest.

Still Furthermore, like the first exemplary embodiment, as shown in FIG.16, the image processing unit includes an R MS-SyncNR unit a209, an R′MS-SyncNR unit a210, a B MS-SyncNR unit a211, and a B′ MS-SyncNR unita212.

As noted above, according to the present exemplary embodiment, the newcolor C1 c may be generated on the basis of the regularly arranged colorfilters C1 a and C1 b. Thereafter, the color C1 c can be interpolatedfor all the pixels on which the color C1 a or C1 b is present.

The present exemplary embodiment can provide the advantages that are thesame as those of the first exemplary embodiment.

While the present invention has been described with reference to theforegoing embodiments, the present invention is not limited thereto.

That is, it should be understood by those skilled in the art thatvarious modifications, combinations, sub-combinations and alterationsmay occur depending on design requirements and other factors insofar asthey are within the scope of the appended claims or the equivalentsthereof.

1-27. (canceled)
 28. A color filter array comprising: a plurality offilters, each having one of a plurality of types of color components;wherein an array pattern in which color filters of the first colorcomponent are arranged in a checker pattern is characterized in that aprobability of occurrence of the first color component stays constant ina desired area; wherein the array pattern in which color filters of atleast two color components are arranged for an arbitrary pixel positionis characterized in that a probability of occurrence of the each colorcomponent is smaller than the probability of occurrence of the firstcolor component in the desired area and a probability of occurrence ofthe one color component of each color component is equal to theprobability of occurrence of another color component.
 29. The colorfilter array of claim 1, wherein the desired area is equal to three bythree pixels.
 30. A color image sensor comprising: a color filter array,having a plurality of filters, each having one of a plurality of typesof color components; wherein an array pattern in which color filters ofthe first color component are arranged in a checker pattern ischaracterized in that a probability of occurrence of the first colorcomponent stays constant in a desired area; wherein the array pattern inwhich color filters of at least two color component are arranged for anarbitrary pixel position is characterized in that a probability ofoccurrence of the each color component is smaller than the probabilityof occurrence of the first color component in the desired area and aprobability of occurrence of the one color component of each colorcomponent is equal to the probability of occurrence of another colorcomponent.